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Symbols and Abbreviations 

The abbreviations in parentheses arc used in selected figures and appendixes. 


F 


ratio of mean square of effect to mean square of associated error term 

P 


probability value 

EPR 

(Epr) 

engine pressure ratio 

Ni 

(Nl) 

fan rotational ratio, r/min 

EGT 

(Egt) 

exhaust gas temperature, °C 

n 2 

(N2) 

compressor rotational speed, r/min 

FF 


fuel flow, lb/hr 

VIB 

(Vib) 

vibration 

PRS 

(Pres) 

pressure, psi 

QL 

(QuanL) 

fuel quantity left, lb (measured in increments of 1000) 

QC 

(QuanC) 

fuel quantity center, 11) (measured in increments of 1000) 

QR 

(QuanR) 

fuel quantity right, lb (measured in increments of 1000) 
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Summary 


Advances in computer technology have increased 
the capability of system designers to generate and 
present information to aid flight crews of commer- 
cial transport aircraft. Aiding flight crews in manag- 
ing in-flight subsystem failures is one area that has a 
high potential benefit, but interface and display tech- 
nology currently may be inadequate to achieve this 
benefit. The purpose of this study, therefore, was 
to identify improved methods for presenting system 
parameter information for the detection of an abnor- 
mal condition and for the identification of the system 
status. Specifically, two workstation experiments us- 
ing static displays were conducted concurrently. The 
primary objective of the first experiment, the Ex- 
pected Value Experiment, was to determine whether 
including cxpected-value-rangc information on tradi- 
tional parameter display formats had any significant 
effect on the ability of a subject to detect abnor- 
mal conditions and identify the status of the affected 
systems. The primary objective of the second ex- 
periment, the Parameter Format Experiment, was to 
determine if using a nontraditional parameter display 
format, which presented relative deviation from the 
expected value, was significantly better than using a 
traditional parameter display format with expected- 
value ranges included for the same task. The inclu- 
sion of cxpected-value-rangc information onto tra- 
ditional parameter display formats had essentially 
no effect on the subjects’ performance for the given 
task in this study. However, the subjective evalua- 
tion results indicate that the subjects show moder- 
ate support for including this information. The non- 
traditional column deviation parameter display 
format (hereafter called the column deviation for- 
mat) resulted in significantly fewer errors when com- 
pared with traditional parameter display formats 
with the expected- value-range information included. 
In addition, error rates for the column deviation for- 
mat remained relatively stable as the scenaiio com- 
plexity increased, wTereas error rates for the tradi- 
tional parameter display formats with expected- value 
ranges increased dramatically. The subjective evalu- 
ation results also indicate that the subjects thought 
that their own performance was better with this col- 
umn deviation format and that they generally pre- 
ferred it over the others. It is therefore recommended 
that the column deviation format be considered for 
display applications that require rapid recognition 
of out-of-tolerance conditions, especially if a large 
number of system parameters must be monitored. 


Introduction 

Background 

Advances in computer technology have increased 
the capability of system designers to generate and 
present information to aid flight crews of commercial 
transport aircraft. Aiding flight crews in managing 
in-flight subsystem failures is one area that has a high 
potential benefit, and research to develop systems 
for aiding failure management is under way for air- 
craft (ref. 1) and for other complex human machine 
interface applications, such as nuclear power plant 
control rooms and computerized hospital operating 
rooms (ref. 2). Appropriate presentation of the in- 
formation generated by these systems, however, is 
very important. Much of this new information may 
have a higher degree of uncertainty than that which 
pilots are currently using because it includes esti- 
mates of the system state in addition to the raw r sen- 
sor data. The uncertainty of this information may 
make current interface and display technology inad- 
equate. This study attempted to resolve one partic- 
ular display issue concerning the presentation of sys- 
tem parameter information for the recognition and 
the identification of abnormal system behavior. 

During the development of the Engine Indication 
and Crew Alerting System (EICAS) displays at The 
Boeing Company during the early 1980's, Boeing en- 
gineers found that the conventional round electro- 
mechanical gauges are actually very good. However, 
Graham (ref. 3) noted that the commercial aircraft 
industry seems to have forgotten “exactly what their 
virtues arc and how their features are used, individ- 
ually and collectively, by the aircrew.” Graham fur- 
ther states that the translation of these displays into 
electronic form is a “deceptively difficult task,” and 
he correctly cautions that although we attempt to de- 
sign electronic displays so that they are better than 
the electromechanical gauges that they replace, we 
must first make sure that the new displays are at 
least as good as the old ones. 

A point that Graham did not address in that pa- 
per, however, is that the conventional gauges arc only 
remarkably good displays on average; that is, they 
do a good overall job for the wide variety of tasks 
for which they are used. Specific tasks may benefit 
greatly from displaying system parameter informa- 
tion in a different form. For example, research has 
recently been performed by Abbott (ref. 4) which 
examined the benefits of presenting engine para- 
meter information in a new format for recognition of 


abnormal engine performance. This new column de- 
viation format displayed the difference between the 
current and the expected value for each parameter. 
Abbott demonstrated that the new format signifi- 
cantly improved recognition of abnormal engine con- 
ditions, especially when the parameter values were 
still within their operating limits. Such was the case 
with the Air Florida Boeing B-737 accident near 
Washington, District of Columbia, in 1982 (ref. 5), 
in which ice impaired the performance of the engine 
pressure ratio (EPR) sensors; this situation misled 
the crew into believing that the engines were develop- 
ing full thrust when they were not. The EPR gauges 
incorrectly indicated normal operation; however, fuel 
flow and exhaust gas temperature (EGT) indications 
were significantly lower than those expected for the 
existing conditions. 

One important purpose of aiding the flight crew 
during fault recognition and system status identifica- 
tion is to increase their situation awareness. Failure 
to understand all the implications of certain system 
failures on the capability of other aircraft systems has 
been cited as a contributing factor in several accident 
and incident cases. These cases include the American 
Airlines, Incorporated, McDonnell Douglas Corpora- 
tion DC- 10 accident at O’Hare airport in Chicago, 
Illinois, in 1979 (ref. 6), in which an engine separation 
during takeoff resulted in an unrecognized asymmet- 
ric leading-edge slat condition, and the China Air- 
lines Boeing B-747 incident enroute to Los Angeles, 
California, in 1985 (ref. 7), in which the autopilot’s 
compensation for loss of thrust on an engine masked 
the asymmetric thrust condition until the autopilot 
was unable to handle it. Another example of the po- 
tentially devastating effects of misunderstanding sys- 
tem failures and their effects was the British Midland 
Airways Boeing B- 737-400 accident near Kegworth, 
Leicestershire, England, in 1989 (ref. 8); in this acci- 
dent, the flight crew mistakenly shut down the wrong 
engine after misinterpreting the symptoms resulting 
from a fan blade separation on the left engine. In 
this case, the flight data recorder proved that the 
instruments were presenting the information neces- 
sary to correctly identify the affected engine. The 
displays did not, however, lead the crew to take the 
correct action. Because the affected engine was still 
producing thrust and the secondary feedback from 
the air-conditioning system suggested that the cor- 
rect action had indeed been taken, the crew’s error 
was not noticed until it was too late to restart the 
good engine before impact with the ground. Thus, 
enhancing the crew members’ awareness of the true 
state of their aircraft should help them not only with 
the current situation but also with any future re- 


lated or unrelated problems that may arise during 
that flight. 

The purpose of this study, therefore, was to iden- 
tify improved methods for presenting system para- 
meter information for the detection of an abnormal 
condition and for the identification of the system sta- 
tus. Specifically, two workstation experiments us- 
ing static displays were conducted concurrently to 
identify particular characteristics that make one for- 
mat preferable to another for displaying system para- 
meter information. The primary objective of the 
first experiment, the Expected Value Experiment, 
was to determine whether including expected-value- 
rangc information on traditional parameter display 
formats had any significant effect on the ability of 
a subject to detect abnormal conditions and iden- 
tify the status of the affected systems. The primary 
objective of the second experiment, the Parameter 
Format Experiment, was to determine if using a non- 
traditional parameter display format, which pre- 
sented relative deviation from expected value, had 
any significant effect on the ability of a subject to 
detect abnormal conditions and identify the status of 
the affected systems compared with traditional para- 
meter display formats with expected-value ranges 
included. Although these two experiments were 
conducted concurrently, they will be treated sepa- 
rately for the purposes of describing the experimental 
method and results. 

Experimental Variables 

In the Expected Value Experiment, the experi- 
mental variables examined were the presence or the 
absence of an expected-value range on a traditional 
parameter display format, the type of traditional 
format used, the level of application of a “display- 
by-exception” philosophy, and the level of scenario 
complexity for each system condition viewed. In 
the Parameter Format Experiment, the experimen- 
tal variables examined were the type of parameter 
display format, the level of application of a display- 
by-exception philosophy, and the level of scenario 
complexity for each system condition viewed. Each 
of these experimental variables is explained below. 
Because the exact form of the expected-value-range 
information in the first experiment depends on the 
traditional parameter display format that it is added 
to, these two variables are explained and discussed 
together. 

Expected- Value Range and Parameter Display 

Format 

As stated above, the primary objective of the Ex- 
pected Value Experiment was to determine wdiethcr 
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including expected- value-range information on tradi- 
tional parameter display formats had any significant 
effect on the ability of a subject to detect abnormal 
conditions and identify the status of the affected sys- 
tems. Here, expected-valuc-rangc information refers 
to the range of values in which a parameter would 
normally be expected to be, given the current op- 
erating conditions and system state. This range is 
normally much smaller than that for all normal con- 
ditions and system states. Traditional parameter dis- 
play formats arc those visual representations of a 
parameter's value which have evolved from electro- 
mechanical dials, gauges, and pointers used in some 
current (and all previous) generations of commercial 
aircraft. Sketches of two of these traditional para- 
meter display formats are shown in figures 1 and 2. 
The circular gauges (fig. 1) are similar to those used 
on the Boeing B-757-200 aircraft (ref. 9) for primary 
engine parameters, and the vertical-scale sliding- 
pointer indicators (fig. 2) are similar to those used 
on the Boeing B-757-200 aircraft for some secondary 
parameters. To maintain consistency with the termi- 
nology used by the subjects, the vertical-scale sliding- 
pointer indicators were referred to throughout this 
report as “bow tie bargraphs” (although they are not 
strictly bargraphs). 

Traditional parameter display formats do not nor- 
mally include any indication of what the ideal value 
of the associated parameter should be for the cur- 
rent conditions. Although fixed caution and warn- 
ing limits arc often shown, these are not changed 
dynamically as the environmental conditions or the 
state of the underlying system changes. For example, 
the ranges of normal values for a turbofan engine s 
oil pressure and temperature increase considerably 
when the engine thrust is increased from idle to a 
takeoff power setting. The traditional caution and 
warning limits for these parameters, however, do not 
account for such changes, and they give no indication 
to the flight crew that these parameters arc expected 
to be within a higher range, given the current thrust 
setting. 

To investigate the potential advantages of includ- 
ing expected-value information, the two traditional 
parameter display formats introduced above, circular 
gauges and bow tie bargraphs, were each enhanced by 
adding the expected- value range. This range was pre- 
sented as a white arc or line that extended 10 percent 
above and below the expected (modeled) value of the 
parameter, given the current condition. Examples 
are shown in figures 3 and 4, respectively, in which 
expected- value-range information has been included 
on the circular gauge format (as an arc segment) and 
on the bow tic bargraph format (as a line segment). 


Both of these traditional parameter display formats 
were included in the experiment, since the specific 
type of format might affect how the subjects use the 
expected- value information. 

Although the parameter formats used were simi- 
lar to those on current commercial aircraft, there was 
one major difference in their operation. For this first 
experiment, we wanted to provide the same visual 
color cues whether the expected- value-range informa- 
tion was included or not. Therefore, since exceeding 
the expected- value range caused the color of the cur- 
rent parameter value and its surrounding box to turn 
either amber or red, this color change was provided 
regardless of whether the expected- value range itself 
was shown or not. Although providing these color 
cues clearly supplies a visual indication that docs 
not currently exist for traditional parameter display 
formats, the color change was included so that any 
differences in subject performance detected with re- 
spect to the expected- value- range information would 
not be confounded by differences in color cues. 

Also as stated above, the primary objective of 
the second experiment, the Parameter Format Ex- 
periment, was to determine if using a nontraditional 
parameter display format that presented relative de- 
viation from expected value had any significant ef- 
fect on the ability of a subject to detect abnormal 
conditions and to identify the status of the affected 
systems compared with traditional parameter display 
formats with expected-value ranges included. The 
nontraditional parameter display format used in the 
second experiment was similar to the Engine Mon- 
itoring and Control System (E-MACS) concept de- 
veloped recently at Langley Research Center (ref. 4) 
as part of a task-oriented display design effort. This 
display is referred to as a column deviation format, 
and it is shown with labels in figure 5 for a single 
display clement. 

The operational concept of the column deviation 
format is considerably different from that of the more 
traditional formats. The horizontal centerline of this 
display represents the expected value of the para- 
meter as provided by a subsystem model, and it is 
shown in green. A solid column of color fill extends 
either up or down from this centerline, thus repre- 
senting cither higher or lower values than those ex- 
pected. The horizontal lines immediately above and 
below the centerline are amber, and they represent 
a 10-percent deviation from the expected (modeled) 
value. When the difference between the actual and 
expected values of the parameter reaches this level, 
the column of color fill reaches one of these lines and 
also turns amber. The horizontal lines above and 
below the amber lines are red, and they represent a 
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15-percent deviation from the expected value. Again, 
when the difference reaches this level, the column of 
color fill reaches one of these lines and turns red. 
One addition that Abbott (ref. 4) made to the basic 
column deviation format is that fixed caution and 
warning limits are represented; this is accomplished 
by ensuring that the height of the column of color 
fill is extended when the actual value is near one of 
those limits. Therefore, even if no difference exists 
between the actual and expected values, if the actual 
value has reached a high caution limit, the column 
of color fill will reach the upper 10-pcrccnt deviation 
line and turn amber. 

Because the expected value of each parameter 
is an inherent part of the column deviation for- 
mat, the two traditional parameter formats used for 
comparison had the expected- value ranges (as de- 
scribed above) added to them. This addition kept 
the information content of the displays similar, even 
though the method of presenting the information was 
different. Specifically, an extra processing step was 
required to determine the relative deviation of a para- 
meter from its expected value when using the tradi- 
tional parameter display formats because the column 
deviation format presented this information directly. 

Levels of Display by Exception 

Based on Allen’s previous work (ref. 9), it was 
concluded that varying the amount of information 
contained in each display by using different levels of 
a display-by-exception philosophy would probably be 
a factor in how well the subjects could perform the 
stated task. The underlying concept of display by 
exception is that only those items requiring imme- 
diate attention should be shown. Therefore, four 
different levels of display by exception were used; 
each represented a different level of application of the 
display-by-exception philosophy. For level 1, all pa- 
rameters for all systems were always displayed, thus 
representing a total absence of display by exception. 
This approach is similar in concept to many current - 
generation cockpits with fixed electromechanical in- 
struments, and it is demonstrated by the sample dis- 
play shown in figure 6 (using circular gauges). For 
level 2, primary engine parameters were always dis- 
played, but parameters for other systems were dis- 
played only when those systems were affected by a 
problem. All parameters for each affected system 
were shown together as a group. This approach 
is similar in concept to the Boeing ETCAS display 
philosophy, and it represents an application of the 
display-by-exception philosophy at the systems level 
(except for the engines). A sample display demon- 
strating this level is shown in figure 7, and it de- 


picts the same system condition used in figure 6. For 
level 3, only parameters of the same type were dis- 
played for affected systems when one or more of those 
parameters were out of tolerance. For example, all 
hydraulic pressure readings were displayed if even one 
was abnormal, but no hydraulic quantity and tem- 
perature readings were shown unless one or more of 
those, too, were abnormal. This approach represents 
an application of the display-by-cxception philosophy 
at the parameter-type level, and it is demonstrated 
by the sample display in figure 8; this display, again, 
depicts the same system condition shown in figures G 
and 7. For level 4 (the last display exception level), 
only out-of-tolerance parameters were displayed for 
the affected systems. This approach represents an 
application of the display- by-exception philosophy at 
the level of each individual parameter reading. Fig- 
ure 9 presents a sample display demonstrating the 
use of this level with the same system condition used 
for figures 6 through 8. A summary of the four dif- 
ferent levels of the display- by-exception philosophy is 
presented in table I. 

Scenario Complexity 

It was also expected, based on Allen’s work 
(ref. 10), that the complexity of the system state 
shown to the subjects would have a significant effect 
on their task performance, so seven different stati- 
cally displayed scenarios with varying levels of com- 
plexity were used. The level of complexity for each 
scenario was manipulated by controlling the number 
of systems (one, two, or three) affected by a problem 
and the number of abnormal parameter readings (sin- 
gle or multiple) within each affected system. Because 
Allen found that increases in the number of affected 
systems resulted in more errors than increases in the 
number of abnormal parameters within each system, 
it was expected that the following scenario descrip- 
tions would result in an approximately monotonic 
increase in complexity level: (1) no abnormal para- 
meters; (2) a single abnormal parameter in a sin- 
gle system; (3) multiple abnormal parameters in a 
single system; (4) a single abnormal parameter in 
each of two systems; (5) multiple abnormal para- 
meters in each of two systems; (6) a single abnormal 
parameter in each of three systems; and (7) multiple 
abnormal parameters in each of three systems. Each 
abnormal parameter in the scenarios had a value that 
was either out of the normal operating range for that 
parameter (i.c., in a caution or warning range) or 
was different from the expected (modeled) value by 
more than 10 percent. Both conditions resulted in 
the same color change. A summary of the seven 
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different levels of scenario complexity is presented in 
table II. 

To make the recognition of parameter conditions 
as realistic as possible, the scenarios used to im- 
plement three of the complexity levels were derived 
from the National Transportation Safety Board ac- 
cident reports (refs. 11 and 12). In addition, nor- 
mal engine values for all seven levels of scenario 
complexity were generated using a Boeing B-737-100 
Pratt & Whitney JT8-D turbofan nonlinear engine 
model that produced thrust values sufficient for a 
Boeing B-737-100 to cruise at Mach 0.70 at 33 000 ft. 
Slight variations from the throttle angles required for 
trimmed flight conditions were used to create the 
engine parameter values for the different scenarios. 
For abnormal engine conditions, the operating ranges 
and limits for the engine parameters were first ob- 
tained from the Boeing Model 737-100 — Operations 
Manual (ref. 13) for the Boeing B-737-100 that was 
equipped with Pratt & Whitney JT8-D turbofan en- 
gines. Values significantly lower and higher than nor- 
mal were then selected as means for low and high val- 
ues, and a corresponding range of variation for each 
mean was also selected. 

The operating ranges and limits for the sub- 
system parameters were also obtained from the 
Boeing manual (ref. 13). Low, normal, and high 
means and corresponding ranges of variation then 
were selected. To obtain the exact parameter val- 
ues used for this experiment for each scenario (except 
for the modeled normal engine conditions), low, nor- 
mal, and high means were randomly varied within 
their ranges using the “minimal standard” pseudo- 
random number generator described by Park and 
Miller (ref. 14). 

One difference between the screen displays tested 
in this study and the displays used in current 
cathode-ray-tube-cquipped (CRT-equipped) aircraft 
is that all system parameters were displayed us- 
ing the same format. For example, if engine para- 
meters were displayed using circular gauges, then all 
oil, hydraulic, electric, and fuel .system parameters 
were also displayed using circular gauges. This sit- 
uation is not generally the case in practice, where 
space constraints force creative compromises. How- 
ever, to maintain consistency and avoid introduc- 
ing yet another experimental factor, different para- 
meter formats were not mixed on the same display 
for this study. Another difference is that current 
CRT-equipped aircraft also use a caution and warn- 
ing system, such as Boeing’s EICAS, w T hich is in- 
tegrated with the parameter display and serves to 
direct the crew’s attention to the appropriate para- 
meters by displaying textual messages. This study 


only investigated the pilot’s ability to use certain 
types of information on the parameter display itself. 

Expected Value Experiment 

The following sections describe the experimen- 
tal method, the experimental hypotheses, the test 
procedures, the data analysis, and the results and 
discussion for the Expected Value Experiment. Be- 
cause this experiment was conducted jointly with the 
Parameter Format Experiment, the relationship be- 
tween these objective results and the subjective data 
are covered jointly with the results of that experi- 
ment in the section of this report entitled “Subjective 
Evaluation Results.” 

Experimental Method 

The Expected Value Experiment examined the ef- 
fects on subject task performance of adding expccted- 
value-range information to traditional circular gauge 
and bow T tie bargraph parameter display formats. 
Twelve test subjects were selected to participate in 
this study. All subjects held an Airline Transport 
Rating on their pilot licenses, and all were “type 
rated” in the Boeing B-737 aircraft. Nine test sub- 
jects were currently employed as flight crew mem- 
bers of a major commercial air carrier, and three 
test subjects were employed as engineering test pi- 
lots. In general terms, each subject’s task was to 
view on a graphics workstation a static display that 
represented a particular combination of the indepen- 
dent variables, and then to clear the screen and ver- 
bally report which aircraft systems were affected by a 
problem and which parameters within those affected 
systems were abnormal. 

The independent variables for the Expected Value 
Experiment included the presence of expected-value- 
range information (included or not included), the 
type of traditional parameter display format (circu- 
lar gauges or bow tie bargraphs), the level of appli- 
cation of the display-by-exception philosophy (lev- 
els 1 through 4, as defined in table I), and the level 
of scenario complexity (levels 1 through 7, as de- 
fined in table II). The experiment was constructed 
as a repeated- measures, full factorial design, in which 
each of the 12 test subjects was tested once on each 
combination of the independent variables (i.e., there 
were no replications within experimental cells). This 
testing resulted in a total of 112 treatments per sub- 
ject for the 2 x 2 x 4 x 7 design. All sequences of 
presentation of the independent variables were either 
counterbalanced or selected randomly to reduce the 
effects of learning and ordering bias on the results. 
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The dependent measures for this experiment were 
the visual, verbal, and total response times, the ac- 
curacy of the subjects’ responses while identifying 
the affected systems and abnormal parameters, and 
the preference data from the subjective evaluation. 
Both visual and verbal response times were obtained 
to detect differences in strategy between pilot sub- 
jects because it was expected that some pilots would 
perform most of the recognition and identification 
tasks while viewing the display and others would per- 
form these steps after clearing the display from the 
screen by relying on short-term memory. The to- 
tal response time, which was an unweighted sum of 
the visual and verbal response times, was also com- 
puted to obtain an overall measure of the task speed. 
The accuracy of the subjects’ verbal responses was 
obtained by counting the number of errors in sys- 
tem identification (i.e., missed/extra systems), the 
errors in parameter identification (i.e., missed/extra 
parameters), the errors in parameter status (i.e., 
caution/warning or high/low status), and the total 
number of errors. Each of these specific error types 
was measured to detect any differences in the types 
of mistakes that were made with the different com- 
binations of treatment conditions. The total number 
of errors, which was an unweighted sum of the other 
error types, was also computed to obtain an overall 
measure of task accuracy. The subject preferences 
as expressed in the subjective evaluation were also 
examined, although they were not evaluated statis- 
tically. The data collected gave general indications 
of the subject’s preferences for or against each level 
of the experimental variables, and they also provided 
feedback about the perceived ease of use. Standard 
workload evaluation techniques, such as the modified 
Cooper-Harper rating scale, were not used because of 
the limited nature of the task in this experiment. 

Experimental Hypotheses 

The experimental hypotheses, in terms of ex- 
pected results from the analysis of the dependent 
measures data, are presented below for the Ex- 
pected Value Experiment. Justifications for these 
expectations are provided where appropriate. 

Expected- Value Range 

It was expected that the inclusion of expcctcd- 
value-range information on both of the traditional 
parameter display formats would lead to a decrease 
in visual response time and a decrease in the number 
of parameter identification and status errors, primar- 
ily because the new information should have made it 


easier to detect and remember the status of the ab- 
normal parameters. It was also expected that the pi- 
lots would favor having this information included on 
the displays. The justification for these hypotheses 
was that expected- value ranges presented the param- 
eter information in a manner more consistent with 
its intended use. Because the experimental task was 
to detect abnormal conditions and report the status 
of the affected parameters, the situation-specific ref- 
erence provided by the expected- value range should 
have made that task easier. 

Parameter Display Formats 

Of the two traditional parameter display formats 
(circular gauges and bow tie bargraphs), the cir- 
cular gauges most closely approximate the electro- 
mechanical round dials that most current pilots have 
used throughout their careers. It was believed to be 
unlikely that the brief training period in this experi- 
ment would have overcome the familiarity associated 
with years of experience. Therefore, it was expected 
that both visual and verbal response times would be 
slightly lower for the circular gauge parameter for- 
mat, that fewer parameter identification and status 
errors in the response would be made, and that the 
pilots would subjectively favor it over the bow tie 
bargraph format. 

Levels of Display by Exception 

It was expected that the levels of display by ex- 
ception which gave the least amount of information 
would lead to a decrease in both visual and verbal 
response times and a decrease in the number of sys- 
tem identification errors made. The justification for 
this hypothesis is that the levels with the least infor- 
mation displayed fewer parameters that were not in- 
dicating abnormal readings. Therefore, scanning the 
display for the affected systems and the parameters 
that were abnormal should have been significantly 
easier because the display was less cluttered. It was 
also expected that the response accuracy would de- 
crease with level 4, since displaying only the abnor- 
mal parameters may remove the visual cues that help 
to distinguish whether those parameters belong to 
the left or the right system. 

Scenario Complexity 

As described earlier, the scenarios were designed 
to present a monotonically increasing amount of com- 
plexity; scenario 1 was the most simple and sce- 
nario 7 was the most complex. It was expected that 
greater scenario complexity would result in signifi- 
cantly higher visual and verbal response times, as 
well as significantly more errors in the response. 
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The issue of fault and status complexity is ex- 
tremely important in the evaluation of display format 
concepts, especially since Allen (ref. 10) found that 
the type of error committed more frequently with 
multiple affected systems is that one or more of these 
systems would be forgotten or ignored. This omis- 
sion indicates a potentially critical lack of situation 
awareness. In fact, this lack of situation awareness 
may cause the crew to respond in an inappropriate 
manner to the current and any future abnormal con- 
ditions. The range of affected systems and fault in- 
dications in the scenarios designed for this study was 
introduced to examine how the number of systems 
and fault indications contributes to complexity for 
the different display concepts examined. However, 
it has also been shown that diagnosis of critical in- 
flight events is more difficult when the necessary sys- 
tem knowledge is not incorporated properly into the 
diagnostic process (ref. 15). This difficulty suggests 
that the “perceived” complexity may be a function of 
the information presentation, the number of affected 
systems and parameters, and the familiarity of the 
pilot with those systems and their interactions. 

Although scenario complexity was included as an 
independent variable in this experiment for the above 
reasons, it is secondary to the main purpose of this 
experiment, and therefore the results and discussion 
relating to complexity are presented in appendix A. 

Test Procedures 

When the test subjects first arrived, they each 
filled out a pilot background questionnaire to record 
information such as years of experience, number of 
flight hours in different aircraft and simulator types, 
and participation in other research studies. All writ- 
ten test materials are included in appendix B. Each 
subject then was asked to read a pilot briefing on 
the current study that explained its goals, described 
the display formats and levels of display by exception 
to be tested (referred to as “presentation styles” in 
the test materials), and explained the actual testing 
procedure. The briefing also instructed each test sub- 
ject to place priority on accuracy, rather than speed, 
during the conduct of the test. 

Before the actual experiment trials began, the 
subjects participated in a brief training exercise on 
the experiment hardware to familiarize them with 
the different treatment conditions and the operation 
of the test program. The training exercise lasted 
approximately 10 minutes, and it presented a single 
scenario (not used for the actual experiment) which 
used each of the different cxpected-value-range con- 
ditions, parameter formats, and levels of display by 


exception. During this time, the subjects were again 
instructed not to sacrifice accuracy for speed, and 
they were trained on the desired verbal protocol for 
the response accuracy measure. 

At this point, the test trials began. For each com- 
bination of levels of the experimental variables, the 
test program first indicated that it was ready to pro- 
ceed. The subjects then pressed the space bar on the 
keyboard, which caused the display to appear on the 
terminal screen. When the subjects finished look- 
ing at the display, they again pressed the space bar, 
which cleared the screen. The test program automat- 
ically calculated the interval between these keyboard 
inputs and recorded it as the visual response time. 
The subjects then verbally identified the affected air- 
craft systems and the condition of each parameter for 
those systems, and then they pressed the space bar 
one last time. The test program calculated the in- 
terval between these keyboard inputs and recorded 
it as the verbal response time. The verbal responses 
of the subjects were manually recorded on a form 
by the experimenter, and each session was also tape 
recorded to preserve the comments of each subject 
and provide verification of the written test results. 

After completing all test trials, which took ap- 
proximately 3 hours, the subjects completed sub- 
jective evaluations to record their impressions about 
their own performance, the ease of use of each dis- 
play, and the display preferences and the reasons for 
these preferences. The subjective evaluations also 
recorded suggestions from the subjects concerning 
how to improve the displays. 

Data Analysis 

Visual, verbal, and total response time data were 
analyzed using a mixed-model analysis of variance 
(ANOVA), with a significance level of 0.05. The 
differences between the means of the effects found 
to be significant in the ANOVA were assessed us- 
ing the Waller-Duncan procedure (ref. 16). Both 
the mixed-model ANOVA and the Waller-Duncan 
tests were computed using the General Linear Mod- 
els (PROC GLM) procedure in the SAS/STAT® 
statistical analysis computer program (ref. 17). 

System identification, parameter identification, 
parameter status, and total error data were also an- 
alyzed using a mixed-model ANOVA followed by the 
Waller-Duncan tests, where appropriate. Although 
frequency data such as the error counts in this exper- 
iment are usually examined using a stratum-adjusted 
Pearson chi-square statistic instead of the ANOVA, 
the more conservative mixed- model ANOVA was 
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used. This substitution was made because the er- 
ror data were so sparse that expected cell frequencies 
in the chi-square analysis were significantly smaller 
than needed (ref. 18). The effect of analyzing the er- 
ror count data using ANOVA, even though the data 
were nonnormally distributed, was that the signifi- 
cance levels were probably slightly higher than those 
reported by the F-tcst in the ANOVA tables (ref. 19). 

For both the response time and error data anal- 
yses, note that the significant main effects that are 
also included in the significant interactions with other 
effects normally will not be examined separately in 
the discussion below. Note also that unless specif- 
ically mentioned, inspection of the few statistically 
significant higher order interactions (three-way inter- 
actions and above) did not reveal any discernible 
patterns, and therefore they are not reported. 

Results and Discussion 

The results of the data analyses for the Expected 
Value Experiment are presented below. Results that 
bear directly on the primary purpose of the experi- 
ment are examined first; results that indicate addi- 
tional findings are examined second. When the re- 
sults of the ANOVA tests are presented, they consist 
of the F-tcst statistic (F, which is the ratio of the 
mean square of the effect to the mean square of the 
associated error term), followed by the probability 
value p. 

Primary Results 

The primary objective of the first experiment, 
as stated in the Introduction, was to determine if 
including expccted-value-rangc information on tra- 
ditional parameter display formats had any signif- 
icant effect on the ability of a subject to detect 
abnormal conditions and identify the status of the 
affected systems. No statistically significant main ef- 
fects or interactions involving expected-value-range 
information were discovered with respect to the vi- 
sual, verbal, or total response time or the num- 
ber of system identification, parameter identification, 
parameter status, or total errors per trial. 

This lack of results for the effects of includ- 
ing expected-value-range information on these tra- 
ditional display formats w T as unexpected. One pos- 
sible explanation for this lack of effects is that more 
than one visual cue was provided which enabled the 
subjects to perform the experimental task. Specifi- 
cally, parameters with values that were outside the 
expected- value range were redundantly coded by a 
color change as described above in the section en- 
title “Experimental Variables.” Since the subjects’ 


task was to identify the abnormal parameters and the 
status of their values, it appears that these subjects 
were primarily using the color change to detect out- 
of-range conditions. Another possible explanation for 
the lack of effects is that there were no wi thin-cell 
replications in the experimental design. This artifact 
of the design could have allowed variations in perfor- 
mance within a single subject to have a detrimental 
effect on the sensitivity of the statistical tests that 
were involved. 

Additional Findings 

Parameter format . Many of the additional sta- 
tistically significant results suggest that the bow tie 
bargraph format was more difficult for the subjects 
to use for this task than the circular gauge format. 
For example, the significant interactions between the 
parameter display format and the level of scenario 
complexity for visual (F G?G g — 2.89, p < 0.05), ver- 
bal (^6,66 = 3.77, p < 0.01), and total (F 6 GG = 4.16, 
p < 0.01) response times, as showm in figures 10, 11, 
and 12, respectively, all indicated that the response 
times for the bow tie bargraphs were significantly 
higher than those for the circular gauges and that 
these differences were largest for the more complex 
scenarios. 

Those results were consistent with expectations, 
since the circular gauge format was much closer in 
appearance to the electromechanical round dials that 
today’s pilots have used for most of their career. The 
bow tie bargraph format, although used in several 
current-generation aircraft, is still relatively new. In- 
terestingly, the fact that the verbal and visual re- 
sponse times were longer for the bow tie bargraph 
format indicated that the subjects may have had 
more difficulty mentally decoding bargraph infor- 
mation as well as visually scanning it. The inter- 
actions with scenario complexity also suggested that 
this familiarity with the circular gauge format was 
most important for the complex scenarios, since less 
excess mental processing capacity was presumably 
available. 

Level of display by exception. The additional 
experimental results that were related to the dif- 
ferences among the four levels of the display by 
exception favored the levels presenting the least 
amount of information with respect to errors in the 
response. These results, however, were inconclu- 
sive with respect to response time. The significant 
(^18,198 = 10.22, p < 0.0001) interaction between the 
level of display by exception and the level of scenario 
complexity for system identification errors, as shown 
in figure 13, indicated that the subjects made fewer 
mistakes when using the levels that presented the 
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least amount of information; however, this occurred 
only for two of the more complex scenarios (i.e., the 
ones with more affected systems and more abnormal 
parameters, such as scenarios 5 and 6). As discussed 
in appendix A, the small number of system identifi- 
cation errors for scenario 7 indicated that it was not 
a good example of the level of scenario complexity 
which it was designed to represent and that scenar- 
ios 5 and 6 were generally perceived by the subjects 
to be more difficult . 

The response time results were less compelling. 
As expected, the significant = 4.50, 

p < 0.0001) interaction between the level of display 
by exception and the level of scenario complexity 
(fig. 14) showed that the visual response times were 
widely separated for the simpler scenarios and that 
the quickest times resulted from those levels of dis- 
play by exception with the fewest gauges to scan. 
However, the mean times for the different levels 
tended to converge as the complexity of the scenario 
increased. This tendency to converge was consis- 
tent with expectations, since the differences between 
the amount of information presented in the various 
levels diminished as the number of abnormal para- 
meters increased. The significant interactions be- 
tween the level of display by exception and the level 
of scenario complexity for verbal (F^ios — 2.49, 
p < 0.01) and total (Ft 8,198 = 3.27, p < 0.0001) re- 
sponse times, as shown in figures 15 and 16, respec- 
tively, showed interesting results for scenario 6, which 
was determined to be the most complex scenario (as 
discussed in appendix A). In this scenario, the lev- 
els of display by exception which displayed the least 
amount of information resulted in the longest ver- 
bal response times, but they had the fewest errors 
(fig. 13). It was unclear why the mean verbal re- 
sponse times increased as the total amount of infor- 
mation displayed decreased, especially since reduc- 
ing the amount of displayed information appeared 
to have reduced the apparent complexity of the sce- 
nario, as evidenced by the fewer system identification 
errors per trial. Although the instructions given to 
the subjects emphasized accuracy over speed, there 
was no obvious bias that should have led them to fol- 
low more or less this advice for the different levels of 
display by exception. 

Parameter Format Experiment 

The following sections describe the experimental 
method, the experimental hypotheses, the test proce- 
dures, the data analysis, and the results and discus- 
sion for the Parameter Format Experiment. Again, 
because this experiment was conducted jointly with 


the Expected Value Experiment, the relationship be- 
tween the objective results and the subjective data 
are covered jointly with the results of that experi- 
ment in the section entitled “Subjective Evaluation 
Results.” 

Experimental Method 

The Parameter Format Experiment examined the 
differences in subject task performance when using 
the circular gauge and the bow tie bargraph pa- 
rameter formats (with expected-value-range infor- 
mation added) and the column deviation format. 
The same 12 test subjects were used for this experi- 
ment as for the Expected Value Experiment, and the 
experimental task was identical. 

The independent variables for the Parameter For- 
mat Experiment included the type of parameter dis- 
play format (circular gauges and bow tie bargraphs 
with expected- value-range information included and 
column deviations), the level of application of the 
display-by-exccption philosophy (levels 1 through 4, 
as defined in table I), and the level of scenario com- 
plexity (levels 1 through 7, as defined in table II). 
This experiment was also constructed as a repeated- 
measures, full- factorial design, in which each of the 
12 test subjects was tested once on each combina- 
tion of the independent variables (i.e., there were no 
replications within experimental cells). This experi- 
ment resulted in a total of 84 treatments per subject 
for the 3x4x7 design. As before, all presentation 
sequences of the independent variables were either 
counterbalanced or selected randomly to reduce the 
effects of learning and ordering bias on the results. 

The dependent measures for the Parameter For- 
mat Experiment were the same as those for the Ex- 
pected Value Experiment: the visual, verbal, and 
total response times; the number of errors in sys- 
tem identification (i.e., missed/extra systems), the 
errors in parameter identification (i.e., missed/extra 
parameters), the errors in parameter status (i.e., 
caution/ warning or high/low status), and the total 
number of errors; and the preference data from the 
subjective evaluation. 

Experimental Hypotheses 

Parameter Display Formats 

The earlier observations concerning the pilots’ 
preexisting familiarity with circular gauges also ap- 
plied for this experiment; however, Abbott’s results 
(ref. 4) showed that error detection rates were much 
higher for a column-deviation-based display than for 
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a combination circular gauge and bow tie bargraph 
display similar to those used in current “glass” cock- 
pits. Therefore, it was expected that this ease of 
recognition would result in both lower response times 
and fewer errors in the response compared with cir- 
cular gauges and bow tie bargraphs (with expected- 
value-range information). It also was expected that 
using circular gauges, because of their familiarity, 
would produce faster response times and fewer errors 
in the verbal response than in the bow tic bargraphs. 
It was unclear, however, because of the novelty of the 
column deviation format, whether the pilots would 
subjectively favor this display over the others. 

Other Factors 

All the expectations described above for the ef- 
fects of presentation style and scenario complexity on 
the subjects’ task performance in the Expected Value 
Experiment also apply to the Parameter Format 
Experiment. 

Test Procedures 

Because the Parameter Format Experiment w r as 
conducted concurrently with the Expected Value 
Experiment, the test procedures used were identical. 

Data Analysis 

Visual, verbal, and total response time data for 
the Parameter Format Experiment were analyzed us- 
ing a mixed-model analysis of variance (ANOVA), 
with a significance level of 0.05. The differences be- 
tween the means of the effects found to be signifi- 
cant in the ANOVA were assessed using the Waller- 
Duncan procedure (ref. 16). As with the Expected 
Value Experiment, both the mixed-model ANOVA 
and the Waller-Duncan tests were computed using 
the General Linear Models (PROC GLM) proce- 
dure in the SAS/STAT® statistical analysis computer 
program (ref. 17). 

System identification, parameter identification, 
parameter status, and total error data were also an- 
alyzed using a mixed-model ANOVA followed by the 
Waller-Duncan tests, where appropriate. For the rea- 
sons given in the section entitled “Data Analysis” for 
the Expected Value Experiment, the error count data 
in the Parameter Format Experiment were also in- 
vestigated using the more conservative mixed-model 
ANOVA instead of the usual chi-square analysis. 

For the response time and error data analyses, 
note that the significant main effects that are also 
included in the significant interactions with other ef- 
fects normally will not be examined separately in the 


discussion below 7 . Note also that unless specifically 
mentioned, inspection of the few 7 statistically signifi- 
cant. higher order interactions (three-way interactions 
and above) did not reveal any discernible patterns 
and therefore are not reported. 

Results and Discussion 

Primary Results 

The primary objective of the second experiment, 
as stated in the Introduction, was to determine if us- 
ing a nontraditional parameter display format (the 
column deviation format), which presented relative 
deviation from expected value, had any significant 
effect on subject performance compared with tradi- 
tional parameter display formats with expected- value 
ranges included. The results relating to errors in the 
responses clearly indicated that the subjects found it 
easier to identify the presence of abnormal parame- 
ters in a particular system when the information was 
presented using the column deviation format com- 
pared with the circular gauge and bow tie bargraph 
formats. As showm in the significant (F \2 ,132 = 3.01, 
p < 0.001) interaction between parameter format and 
scenario complexity for the mean number of system 
identification errors in figure 17, far fewer errors were 
made identifying affected systems using the column 
deviation format, especially for the most complex 
scenarios (i.e., scenarios 5, 6, and 7). This reduc- 
tion of errors suggested that presenting parameter 
information using the column deviation format re- 
duced the apparent task complexity. This result was 
consistent with expectations, since the column devi- 
ation format wrns expressly designed for detecting ab- 
normalities during parameter monitoring tasks. This 
finding also agrees with the results found by Abbott 
in reference 4. 

In addition, the response time results suggested 
that the subjects could perform the task more quickly 
using the circular gauge and column deviation for- 
mats than they could with the bow 7 tie bargraph 
format, especially for the most complex scenar- 
ios. For example, the significant interactions be- 
tween parameter format and scenario complexity for 
the mean visual (Fi2 ; i32 = 2.25, p < 0.05), verbal 
(Fi2,i32 = 2-88, p < 0.01) , and total (F 12 ,i 32 = 2.99, 
p < 0.001) response times, showm in figures 18, 19, 
and 20, respectively, indicated that it usually took 
the subjects as long or longer to respond both visu- 
ally and verbally to the scenarios when using the bow 
tie bargraph format than when using either the cir- 
cular gauge or column deviation format. The largest 
differences among the parameter formats occurred 
in the more complex scenarios, as expected, but the 
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similarity in response times for the more simple sce- 
narios prevented the main effect of parameter for- 
mat (across all levels of complexity) from reaching 
statistical significance. 

Many other issues beyond those examined in this 
particular study, however, still remain concerning the 
use of column deviation formats for parameter pre- 
sentation. For example, the ability to rapidly scan 
column deviation display formats may be affected if 
the parameter information is distributed among the 
different elements of a system schematic. Moreover, 
it was shown by Kieras (ref. 20) that such a distribu- 
tion of parameter data within a system schematic (so 
that each parameter is in close proximity to its re- 
lated component) is the preferred method to present, 
this information for diagnosis. The pictorial system 
status format of Summers (ref. 21) also distributed 
the parameter information within the schematic, and 
this format was shown to be better than a textual 
format for abnormal condition detection and system 
identification. It is therefore recommended that ad- 
ditional studies be performed to determine if differ- 
ent methods of presenting system parameter informa- 
tion, depending on the set of tasks to be performed, 
will enhance overall performance. 

Additional Findings 

The observations made with respect to differences 
in performance among the four levels of display by ex- 
ception in the Expected Value Experiment were also 
supported by the additional results of this experi- 
ment: the error results generally favored the levels 
presenting the least information, but the response 
time results were inconclusive. As in the first ex- 
periment, the subjects usually made fewer mistakes 
with the levels of display by exception which pre- 
sented the least amount of information. The sig- 
nificant (F\g,i 98 = 6.92, p < 0.0001) interaction be- 
tween the level of exception and the level of scenario 
complexity for the mean number of system identifi- 
cation errors, as depicted in figure 21, showed that 
the greatest contribution to the decrease in system 
identification errors as information was removed from 
the display occurred during one of the most com- 
plex scenarios (i.e., scenario 6). However, the signif- 
icant (Fi 8 ,i98 = 2.27 , p < 0.01) interaction between 
the level of display by exception and the level of sce- 
nario complexity for the mean number of parameter 
status errors, as shown in figure 22, does not offer 
evidence that reducing the amount of displayed in- 
formation reduces the perceived complexity of the 
scenario. 

As in the first experiment, the response time 
results were statistically significant, but they 


were inconclusive. For example, the significant 
(^18 198 = 2.18, p < 0.01) interaction between the 
level of display by exception and the level of scenario 
complexity for mean visual response time, seen in 
figure 23, followed the pattern discovered in the first 
experiment. The visual response times for the four 
levels of display by exception tended to converge as 
the scenarios became more complex because the dif- 
ferences in amount of information presented among 
the four levels decreased as the number of abnormal 
parameters increased. 

Subjective Evaluation Results 

A subjective evaluation in the form of a struc- 
tured questionnaire (the “Subjective Evaluation 
Parameter Presentation Study, as seen in appen- 
dix B) was administered to the subjects after all 
experimental trials for the Expected Value Experi- 
ment and the Parameter Format Experiment were 
completed. In questions 1 through 12, the subjects 
were asked to evaluate their own performance for 
each parameter format, expected- value-range status, 
and level of display by exception (referred to as “pre- 
sentation style 7 ’ in the questionnaire), with regard 
to how easily and quickly they could detect out- 
of-tolcrance conditions and remember these condi- 
tions for the verbal response. Summaries of the re- 
sponses to questions 1 through 12 on the subjective 
evaluation arc presented in figures 24 through 35, 
respectively. In questions 13 and 14, the subjects 
were asked to rank the three combinations of format, 
expected-value range, and level of display by excep- 
tion that they liked the best and the three combina- 
tions that they liked the least. Summaries of these 
responses are presented, grouped by test condition, 
in figures 36 through 40. In questions 15 and 16, the 
subjects were asked to provide the reasons for their 
rank selections from questions 13 and 14. Finally, 
in questions 17 through 19, the subjects were asked 
for general comments on how the display concepts 
could be improved. Summaries of all the responses 
to questions 15 through 19 are not presented, but 
selected comments are incorporated in the following 
discussion. 

For questions 1 through 12 of the subjective eval- 
uation, average responses to each question for all 
12 tost subjects were computed. Because these ques- 
tions all dealt with deviations from a null response, 
the average responses were calculated by assigning 
specific values to each of the blocks on the rating scale 
used to answer the questions. The extreme left-hand 
block, which corresponded to answers such as “very 
slowly” or “very difficult,” and the extreme right- 
hand block, which corresponded to answers such as 
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‘Very quickly” or “very easy,” were assigned values 
of -2 and 2, respectively. The blocks just left of 
center and just right of center, which corresponded 
to answers such as “somewhat slowly” or “some- 
what difficult,” and “somewhat quickly” or “some- 
what easy,” respectively, were assigned values of -1 
and 1. The center block corresponded to the an- 
swer “neither,” and it was assigned a value of 0. 
An average response for all 12 test subjects that ex- 
ceeded 0.75 was considered moderate support for as- 
suming a nonnull response, while an average response 
that exceeded 1.50 was considered strong support. 

The results from questions 13 and 14 of the sub- 
jective evaluation were combined to provide a single 
average response for each condition. The mean re- 
sponse was calculated using a value ranging from —3 
for the least-liked response from question 14, to 3 
for the best-liked response from question 13. No null 
response existed for these questions. The average 
responses were considered to be different from one 
another only if these differences exceeded 0.75. 

Parameter Display Formats 

The responses to the first four questions of 
the subjective evaluation, summarized in figures 24 
through 27, indicated that many of the pilots be- 
lieved their performance was better using the column 
deviation format compared with the circular gauge 
or bow tie bargraph formats. Specifically, the re- 
sponses to questions 1, 3, and 4 all strongly indi- 
cated that the subjects believed that when using the 
column deviation format, they could more quickly 
detect out-of- tolerance parameters, more quickly de- 
termine whether those out-of-tolerance parameters 
were too high or too low, and more easily remem- 
ber (for the verbal response) whether those out-of- 
tolerance parameters were too high or too low. The 
responses to question 1 also moderately indicated 
that the subjects believed that when using the circu- 
lar gauge parameter format they could more quickly 
detect out-of-tolerance parameters. 

In addition, as seen in figures 38, 39, and 40, re- 
spectively, the combined average responses to ques- 
tions 13 and 14 for circular gauges and bow tie 
bargraphs (with expected- value- range information) 
and column deviations moderately indicated that the 
subjects preferred column deviations over both cir- 
cular gauges and bow tic bargraphs. Interestingly, 
when the responses were subdivided by level of dis- 
play by exception, it was seen that only the column 
deviation format received positive feedback all four 
styles. The combined average responses for circular 
gauges and bow tie bargraphs (with expected-value- 


rangc information) suffered most from negative feed- 
back for level of the display by exception in which all 
parameters were displayed all the time. All three of 
the parameter formats, however, were strongly pre- 
ferred when displayed using level 3 of the display by 
exception, in which all parameters of the same type 
were displayed for affected systems when one of those 
parameters was out of tolerance. 

Comments from the subjects also indicated that, 
in general, they preferred the column deviation for- 
mat. Remarks such as “easy to scan, analyze, and re- 
member,” “more logical,” “by far the easiest to read 
and interpret,” “seemed to make abnormal indica- 
tions much more prominent,” and “jumps out and 
grabs your attention” were common. Several pilots 
also commented, however, that they preferred circu- 
lar gauges and bow tie bargraphs because of “clarity 
and familiarity” and because they are “common on 
most aircraft that I operate” and “what I ain most 
trained to use.” 

These results usually agreed with the objective 
results of the Parameter Format Experiment because 
the use of the column deviation format resulted in 
significantly fewer errors in system identification; 
fewer errors meant that the pilots were having less 
difficulty remembering which systems were affected 
by faults. Also, response times with the column 
deviation format were essentially the same as with 
the circular gauge format, and both these formats 
resulted in faster responses than with the bow tie 
bargraph format, even though the pilots were more 
familiar with the circular gauge and bow tie bargraph 
formats. 

Expected- Value Range 

The responses to questions 5 through 8 of the 
subjective evaluation, summarized in figures 28 
through 31, respectively, indicated that many of the 
pilots believed that the expected-value ranges helped 
them more quickly detect problems. Specifically, 
the responses to questions 5 and 7 moderately indi- 
cated that with the expected- value ranges included, 
the subjects believed they could more quickly de- 
tect out-of- tolerance parameters and more rapidly 
determine whether these parameters were too high 
or too low. As seen in the responses to questions 6 
and 8, however, the pilots did not in general be- 
lieve that expected- value ranges helped them remem- 
ber the scenario conditions any better for the verbal 
response. 

The combined average responses from ques- 
tions 13 and 14 of the subjective evaluation, for cir- 
cular gauges and bow tie bargraphs both without 
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(figs. 36 and 37) and with (figs. 38 and 39) expected- 
value ranges included, offered strong support that the 
subjects preferred having the expected-value-range 
information on the display. This preference is seen 
particularly when the combined average responses 
for circular gauges without expected-value ranges 
(-1.364) and with expected- value ranges (0.733) are 
compared because this is the parameter format with 
which the pilots were most familiar. 

Comments from the pilots also indicated that 
they generally preferred having the expected-value- 
range information included on the traditional param- 
eter display formats. One pilot “liked [the] expected- 
value range (but didn’t realize at first that I was using 
it).” Another pilot observed that “expected- value 
ranges would be a welcome addition... hard range 
marks arc of limited value in a dynamic environ- 
ment.” A third pilot, however, remarked that adding 
expected- value ranges resulted in “too much clutter” 
and made it harder to “extract what’s important. 

The preferences of the pilots for the expected- 
valuc-range information were not supported by the 
objective data. However, some of the subjective com- 
ments indicated that the pilots were extrapolating 
the usefulness of this information into situations and 
conditions beyond those used in these experiments. 
Other comments suggested that additional training 
and operational experience may be required before 
the information is used to its full potential. 

Levels of Display by Exception 

The responses to questions 9 through 12 of 
the subjective evaluation, summarized in figures 32 
through 35, indicated that many of the pilots be- 
lieved their performance was better using the two lev- 
els of display by exception that presented the least 
amount of information. Specifically, the responses 
to all of these questions moderately indicated that 
the subjects believed that, using levels 3 and 4, they 
could more quickly detect and more easily remember 
(for the verbal response) out-of-tolerance parameters 
and more quickly determine and more easily remem- 
ber whether those out-of-tolerance parameters were 
too high or too low. 

The responses from questions 13 and 14 of the 
subjective evaluation, however, did not always indi- 
cate support for those beliefs. Although the level 
presents- the most information (level 1) received 
cdback for four of the five test conditions 
figs. 36 through 39), level 4, which pre- 
cast information, also received some nega- 
t (as seen in figs. 36 through 38). Level 3 
\nost positive feedback in four of the five 


conditions. For the column deviation format (fig. 40), 
the subjects indicated a higher preference for level 1 
than for level 4; however, level 3 again received the 
most positive feedback. 

Comments from the subjects indicated that they 
were divided concerning whether the levels of display 
by exception that showed less information were really 
better than the others. For example, several of 
the pilots thought that level 1, which displayed all 
the parameters all of the time, was “too busy or 
“too cluttered.” Other pilots also thought that “the 
parameters should not be displayed unless there is a 
problem.” However, many pilots also wanted other 
parameters shown for comparison. Remarks such as 
“I like being able to compare with the other similar 
systems” and “I like having all parameters up if the 
display is not too busy” were common. Even these 
pilots, though, tended to like level 3, which displayed 
all similar gauges for a system even if only one was 
out of tolerance. One pilot stated that “...style 
[level] 3 was [the] best decluttcr mode.” 

These findings agree with the objective results of 
the Expected Value Experiment and the Parameter 
Format Experiments because levels 3 and 4 of the 
display by exception generally had fewer errors and 
quicker response times across different parameter 
formats and levels of scenario complexity. 

Conclusions and Recommendations 

The inclusion of expected-value information onto 
traditional parameter display formats had essentially 
no effect on the performance of the subjects for 
the given task in this study. However, the subjec- 
tive evaluation results indicated that the subjects 
showed moderate support for including this infor- 
mation. Because specific comments also indicated 
that the subjects, all experienced pilots, were men- 
tally extrapolating the usefulness of this information 
into situations other than those encountered in this 
study, it is recommended that display designers con- 
sider including this type of information for tasks that 
include detecting system abnormalities that develop 
over a period of time. 

Use of the nontraditional parameter display for- 
mat implemented in this study, a column deviation 
format based on Abbott’s (ref. 4) Engine Monitor- 
ing and Control System (E-MACS) concept, resulted 
in significantly fewer errors in system identification 
when compared with traditional parameter display 
formats with expected- value information included. 
In addition, use of the column deviation format ap- 
pears to have resulted in less perceived complexity 
for this task because system identification error rates 


remained relatively stable as the scenario complexity 
increased, whereas error rates for the traditional pa- 
rameter display formats with expected- value ranges 
increased dramatically. The subjective evaluation re- 
sults also indicate that the subjects thought that 
their own performance was somewhat better with 
this parameter format and that they generally pre- 
ferred it over the others. It is therefore recommended 
that the column deviation format be considered for 
display applications that require rapid recognition 
of out-of-tolerance conditions, especially if a large 
number of system parameters must be monitored. 

The effects of applying four different levels of the 
display- by-exception philosophy were mixed. Ver- 
bal response times were reduced as more extrane- 
ous information was removed from the display, but 
verbal response times for the most complex scenario 
were increased. Interestingly, the number of system 
identification errors was significantly reduced, with- 
out causing the expected increase in parameter sta- 
tus errors. Reducing the amount of information on 
the display also appeared to have reduced the per- 


ceived complexity of the scenario, since the number 
of system identification errors for the more complex 
scenarios decreased significantly as the amount of in- 
formation decreased. In addition, the subjective re- 
sults indicated that the subjects believed that the 
levels of display by exception with the least infor- 
mation actually improved the speed with which they 
could identify out-of-tolerance parameters, although 
they still had concerns about being able to see all the 
parameters when they needed to see them. These 
findings suggest that display designers should care- 
fully evaluate the impact of including nonessential 
information on displays that are used primarily for 
rapid recognition of out-of-tolerance system condi- 
tions. When such displays are used for other system 
tasks, however, the subjective results suggest that all 
the information that supports those tasks should be 
presented. 


NASA Langley Research Center 
Hampton, VA 23681-0001 
January 3, 1994 
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Table I. Summary of Levels 1 Through 4 of Display by Exception 


Level of display 
by exception 

Description 

1 

All parameters for all systems are always displayed 

2 

Primary engine parameters are always displayed, but parameters for other 
systems are shown only when those systems are affected by problem 

3 

Only parameters of same type are displayed for affected systems when 
one or more is out of tolerance 

4 

Only out-of-tolerance parameters are displayed 


Table II. Summary of Levels 1 Through 7 of Scenario Complexity 


Level of scenario 
complexity 

Description 

1 

No abnormal parameters are displayed 

2 

Single abnormal parameter is displayed in single system 

3 

Multiple abnormal parameters are displayed in single system 

4 

Single abnormal parameter is displayed in each of two systems 

5 

Multiple abnormal parameters are displayed in each of two systems 

6 

Single abnormal parameter is displayed in each of three systems 

7 

Multiple abnormal parameters are displayed in each of three systems 
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Figure 2. Example of bow tie bargraph parameter format. 




Figure 3. Example of circular gauge parameter format with expected-value-rangc information included. 
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Figure 4. Example of bow tie bargraph parameter format with cxpected-value-range information included. 


Upper Limit Line (Red) ► 

+ 15% Deviation Line (Red) ► 

+10% Deviation Line (Amber) ► 

Centerline (Green) ► 

-10% Deviation Line (Amber) ► 

-15% Deviation Line (Red) ► 

Lower Limit Line (Red) ► 


79.1 ◄ Parameter Value 



N2 


Parameter Label 


Figure 5. Example of column deviation format with explanatory labels. 
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Figure 7. Level 2 of display by exception (in which primary 
parameters for other systems are only displayed when system 


engine parameters are always displayed, but 
is affected by fault); circular gauge parameter 


format is used. 
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Figure 8. Level 3 of display by exception (in which all parameters of same type are displayed if one or more is 
out of tolerance); circular gauge parameter format is used. 


20 






Figure 9. Level 4 of display by exception (in which only out-of-tolerance parameters are displayed); circular 
gauge parameter format is used. 
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Figure 10. Mean visual response time versus level of scenario complexity for each parameter format for Expected 
Value Experiment. 



Figure 11. Mean verbal response time versus level of scenario complexity for each parameter format for 
Expected Value Experiment. 
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Figure 12. Mean total response time versus level of scenario complexity for each parameter format for Expected 
Value Experiment. 



Figure 13. Mean number of system identification errors per trial versus level of scenario complexity for each 
level of display by exception for Expected Value Experiment. 
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Figure 14. Mean visual response time versus level of scenario complexity for each level of display by exception 
for Expected Value Experiment. 



Level of Scenario Complexity 

Figure 15. Mean verbal response time versus level of scenario complexity for each level of display by exception 
for Expected Value Experiment. 
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Level of Scenario Complexity 

Figure 16. Mean total response time versus level of scenario complexity for each level of display by exception 
for Expected Value Experiment. 



Figure 17. Mean number of system identification errors per trial versus level of scenario complexity for each 
parameter format for Parameter Format Experiment. 
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1 2 3 4 5 6 7 

Level of Scenario Complexity 

Figure 18. Mean visual response time versus level of scenario complexity for each parameter format for 
Parameter Format Experiment. 



Level of Scenario Complexity 

Figure 19. Mean verbal response time versus level of scenario complexity for each parameter format for 
Parameter Format Experiment. 
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Level of Scenario Complexity 

Figure 20. Mean total response time versus level of scenario complexity for each parameter format for Parameter 
Format Experiment. 


Mean Number 
of System 
Identification 
Errors per Trial 



Figure 21. Mean number of system identification errors per trial versus scenario for each level of display by 
exception for Parameter Format Experiment. 
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Figure 22. Me mii numbci of parameter status errors per trial versus level of scenario complexity for each level 
of display by exception for Parameter Format Experiment. 



Level of Scenario Complexity 

Figure 23. Mean visual response time versus level of scenario complexity for each level of display by exception 
for Parameter Format Experiment. 
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Figure 24. Responses to question 1 of subjective evaluation (concerning parameter formats): “How quickly 
could you detect out-of-tolerance parameters?” 
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Very Difficult Very Easy 


Figure 25. Responses to question 2 of subjective evaluation (concerning parameter formats): “How easy was 
it to remember which parameters were out of tolerance?” 
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Very Slowly Very Quickly 

Figure 26. Responses to question 3 of subjective evaluation (concerning parameter formats): 
could you determine whether the out-of-tolerance parameters were too high or too low?” 
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Very Difficult Very Easy 

Figure 27. Responses to question 4 of subjective evaluation (concerning parameter formats): “How easy was 
it to remember whether the out-of-tolcrance parameters were too high or too low?” 
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Figure 28. Responses to question 5 of subjective evaluation (concerning expected- value-range information): 
“How much more quickly or slowly could you detect out-of-tolerance parameters when expected value 
ranges were added?” 



Much More Much 

Difficult Easier 

Figure 29. Responses to question 6 of subjective evaluation (concerning expectcd-value-range information): 
“How much easier or more difficult was it to remember which parameters were out of tolerance when 
expected- value ranges were added? ’ 
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Figure 30. Responses to question 7 of subjective evaluation (concerning expected- value-range information): 
“How much more quickly or slowly could you determine whether the out-of-tolerance parameters were too 
high or too low when expected- value ranges were added?” 
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Figure 31. Responses to question 8 of subjective evaluation (concerning expected- value-range information): 
“How much easier or more difficult was it to remember whether the out-of- tolerance parameters were too 
high or too low when expected- value ranges were added?” 
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Figure 32. Responses to question 9 of subjective evaluation (concerning level of display by exception): “How 
quickly could you detect out-of-tolerance parameters?” 
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Figure 33. Responses to question 10 of subjective evaluation (concerning level of display by exception): 
easy was it to remember which parameters were out of tolerance?" 
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Figure 34. Responses to question 1 1 of subjective evaluation (concerning level of display by exception; 
quickly could you determine whether the out-of-tolerance parameters were too high or too low. 


: “How 
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Very Difficult Very Easy 

Figure 35. Responses to question 12 of subjective evaluation (concerning level of display by exception): “How 
easy was it to remember whether the out-of-tolerance parameters were too high or too low?” 
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Total Number of Responses = 2 
Average Response = 1 .000 


Total Number of Responses = 0 
Average Response = N/A 


Total Number of Responses = 4 
Average Response = -1.750 


-2 


Best Liked 


Fieurc 36 Responses for circular gauges without expected- value-range information to questions 13 and 14 of 
subjective evaluation: “Please select the three combinations of display format, expected-value range, and 
presentation style [level of display by exception] that you like the least/best.” Total number of responses 
across all four levels for this condition was 11, and combined average response was -1.364. 


39 


6 

5 

4 

Number of 
Responses for 3 
Level 1 

2 

1 

0 


Total Number of Responses = 5 
Average Response = -2.600 



Least Liked 


Best Liked 


Figure 37. Responses for bow tic bargraphs without expected- value-range information to questions 13 and 14 
of subjective evaluation: “Please select the three combinations of display format, expected-value range, and 
presentation style [level of display by exception] that you like the least/best.” Total number of responses 
across all four levels for this condition was 14, and combined average response was —1.571. 
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Figure 38. Responses for circular gauges with expected- value-range information to questions 13 and 14 of 
subjective evaluation: “Please select the three combinations of display format, expected-value range, and 
presentation style [level of display by exception] that you like the lcast/best. Total number of responses 
across all four levels for this condition was 15, and combined average response was 0.733. 
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Figure 39. Responses for bow tie bargraphs with expected- value-range information to questions 13 and 14 of 
subjective evaluation: “Please select the three combinations of display format, expected- value range, and 
Presentation style [level of display by exception] that you like the least/best.” Total number of responses 
across all four levels for this condition was 11, and combined average response was 0.273. 
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Figure 40. Responses for column deviations to questions 13 and 14 of subjective evaluation: “Please select 
the three combinations of display format, expected- value range, and presentation style [level of display by 
exception] that you like the least/best.” Total number of responses across all four levels for this condition 
was 21, and combined average response was 1.095. 
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Appendix A 

Results and Discussion of Scenario 
Complexity 

The effects of scenario complexity on response 
time and accuracy were essentially as predicted for 
both the Expected Value Experiment and the Para- 
meter Format Experiment; that is, increasing the 
complexity led to longer response times and more 
errors. In addition, increasing the complexity of the 
scenarios also helped to highlight differences between 
display formats which may otherwise have remained 
obscured by uniformly good performance on the task. 
Both experiments, however, indicated that although 
the experimental measures that were employed def- 
initely detected changes in performance caused by 
scenario complexity, the set of dimensions originally 
used to define complexity for this study was ap- 
parently incomplete. This incompleteness was due 
to the level of scenario complexity not being per- 
ceived by the subjects as monotonically increasing 
with scenario number, as intended. For instance, the 
interactions between parameter format and level of 
scenario complexity for visual, verbal, and total re- 
sponse times in the Expected Value Experiment, seen 
in figures 10, 11, and 12, respectively, all showed that 
regardless of the parameter format, the subjects took 
longer to respond to situations that were more com- 
plex. The interactions between level of display by ex- 
ception and level of scenario complexity for mean ver- 
bal and total response times in the Expected Value 
Experiment, however, as seen in figures 15 and 16, 
respectively, also suggested that scenario 6 was sig- 
nificantly different from either scenario 5 (which was 
designed to be less complex) and scenario 7 (which 
was designed to be more complex). 

The most striking evidence of the differences 
in perceived complexity between scenario 6 and 
scenarios 5 and 7 was seen in the error results. 
The significant differences in mean number of para- 
meter identification = 13.94, p < 0.0001), 

parameter status (F 6j66 = 4.53, p < 0.001), and to- 
tal (Tg eg = 18.14, p < 0.0001) errors for each of the 
seven scenarios in the Expected Value Experiment 
arc shown in figures Al, A2, and A3, respectively. 
For the parameter identification and the total num- 
ber of errors (figs. Al and A3), a Waller- Duncan test 
showed that the means for the seven levels of scenario 
complexity were divided into three clusters. The first 


cluster contained only scenario 6; the second con- 
tained scenarios 5 and 7; and the third contained the 
rest. For the parameter status errors (fig. A2), the 
Waller- Duncan test showed that the means were di- 
vided into only two overlapping clusters, with scenar- 
ios 6, 5, 7, and 4 in the first cluster and scenarios 4, 
3, 2, and 1 in the second cluster. Taken together, 
these results indicate that scenario 6 was seen to be 
much more difficult than the others; this difficulty 
extended to the point that the mean total number of 
errors per trial for scenario 6, as seen in figure A3, 
was greater than 1. 

The error results for the Parameter Format 
Experiment were similar. The significant differ- 
ences in mean number of parameter identifica- 
tion (F 6 ge = 11.79, p < 0.0001), parameter status 
(F 5,66 = 2.81, p < 0.05), and total (F 6) 66 = 17.66, 
p < 0.0001) errors for each of the seven scenarios 
are shown in figures A4, A5, and A6, respectively. 
For the parameter identification errors (fig. A4), a 
Waller-D uncan test showed that the means for the 
seven levels of scenario complexity were divided into 
four clusters, three of which overlapped. The first 
cluster contained only scenario 6; the second con- 
tained scenarios 5 and 7; the third contained scenar- 
ios 7 and 4; and the fourth contained scenarios 4, 2, 
3, and 1. For the parameter status errors (fig. A5), 
the means were divided into only two overlapping 
clusters, with scenarios 7, 6, 5, and 4 in the first 
and 5, 4, 3, 2, and 1 in the second. Finally, the 
means for the total number of errors (fig. A6) were 
divided into three disjoint clusters. The first cluster 
contained scenario 6; the second contained scenar- 
ios 5 and 7; and the third contained the rest. As 
in the Expected Value Experiment, scenario 6 was 
apparently the most difficult. 

Further inspection of the descriptions of the sce- 
narios used for this study indicated that scenario 7 
was probably seen by the subjects as less complex 
than scenario 6 for at least two reasons. First, the 
description of scenario 7 lists the left and right en- 
gines as two separate systems, when in fact the sub- 
jects appeared to have considered them together, es- 
pecially since they exhibited identical behavior in this 
case. Second, all abnormal parameters in the three 
affected systems (left engine, right engine, and left 
electrical system) for scenario 7 had the same sta- 
tus; that is, the parameters were all low and in the 
warning region. This regularity apparently had the 
effect of creating a situation that was much less com- 
plex to grasp, even though scenario 7 had a total of 
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nine abnormal parameters, while scenario 6 had only 
three. 

Based on the scenario complexity results of the 
Expected Value Experiment and the Parameter For- 
mat Experiment , a new set of dimensions to define 
the complexity of a statically viewed scenario should 
include the following: (1) the number of affected sys- 
tems of different types; (2) the number of abnormal 


parameters in each affected system; (3) the number 
of differences in status (caution or warning range) for 
the abnormal parameters, both within and between 
affected systems; and (4) the number of differences in 
qualitative value (too high or too low) for the abnor- 
mal parameters, both within and between affected 
systems. Additional dimensions, such as the rate of 
change for each parameter, will clearly be necessary 
if dynamic scenarios are used. 
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Figure Al. Mean number of parameter identification errors per trial versus level of scenario complexity for 
Expected Value Experiment. 
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Figure A2. Mean number of parameter status errors per trial versus level of scenario complexity for Expected 
Value Experiment. 
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Figure A3. Mean total number of errors per trial versus level of scenario complexity for Expected Value 
Experiment. 


Mean Number 
of Parameter 
Identification 
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Figure A4. Mean number of parameter identification errors per trial versus level of scenario complexity for 
Parameter Format Experiment. 
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Figure A5. Mean number of parameter status errors per trial versus level of scenario complexity for Parameter 
Format Experiment. 



Figure A6. Mean total number of errors per trial versus level of scenario complexity for Parameter Format 
Experiment. 


48 



Appendix B 


Written Test Materials 

The written materials for this experiment included the Pilot Background Questionnaire, the Pilot Briefing: 
Parameter Presentation Study, the Trial Response Form (which was filled out by the researcher during the 
experiment), and the Subjective Evaluation: Parameter Presentation Study. All materials are included here 
exactly as seen by the subjects. Please note that the experimental variable “level of display by exception” was 
referred to as “presentation style” in these materials and during all interactions with the subjects. 

PILOT BACKGROUND QUESTIONNAIRE 

Name : 

Address: 

Home Phone : 

Other Phone: {if applicable) : 

Employer: 

Rank (Position) : 

Years Flying Commercial: Flying Military: _ — _ 

Approximate Total Time (Hours) : _ 

Flight Experience - Type of aircraft and approximate hours (No GA) . 

Hours in Type 


Simulator Experience — Type of simulator and approximate hours. 
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PILOT BACKGROUND QUESTIONNAIRE 


Would you be interested in participating in (check each that applies) : 

tests which require flying the NASA B-737? (must be 737 

qualified) 

tests which require flying one of the NASA simulation 

facilities? 

evaluations of new displays and flight deck systems? (no 

flying) 

interviews and studies relating to aircraft safety, 

automation, etc.? 

Have you ever participated in a NASA research project before? 

If YES, then either briefly describe the test /interview or give the 

researcher's name who conducted the test: 


How much lead time will you require for scheduling appointments? 


Please list any colleagues who may be interested in participating in 
NASA studies (include address and phone number, if known) : 


Please return this form to: 

Pilot Questionnaire 
Vehicle Operations Research Branch 
Mail Stop 156A 

NASA Langley Research Center 
Hampton, VA 23665 

THANK YOU FOR YOUR COOPERATION! 
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PILOT BRIEFING 

PARAMETER PRESENTATION STUDY 


Introduction 

The purpose of this study is to identify improved methods for 
p resen ting system parameter information to permit faster and better 
recognition of system faults and status. Five different display formats 
will be tested. Each of these five formats will have four different 
styles of "display-by-exception 11 applied to them, so that the total 
amount of information will be different depending on what has gone wrong. 

The study will be conducted in five parts, one for each of the five 
display formats. Within each of these parts, each of the four display- 
by-exception styles will be used with a series of aircraft system 
states. Your tasks will be to determine which systems are affected in 
each state that you view and what the status is of the parameters for 
those affected systems. 

Please remember that this study is not attempting to develop new 
instrument display formats for immediate commercial aircraft use. 

Rather, it is an attempt to develop guidelines for future development of 
display formats for system parameter information. This is an important 
dist inct ion . 

Display Formats 

The five parameter display formats you will see, (1) circular 
gauges, (2) bargraphs, (3) circular gauges with expected value range, 

(4) bargraphs with expected value range, and (5) column deviations, are 
shown in Figures 1 through 5 [figs. B1 through B8] . Basically, they are 
all variations on the following form: 


Engines 

Oil 

Electrical 

Hydraulic 

Fuel 



Note that there are really only three different types of display 
elements: circular gauges, bargraphs, and column deviations. To create 

formats 3 and 4, a new piece of information, an expected value range, 
has been added to the circular gauges and bargraphs. This expected 
value range sweeps out a range of ±10% around the expected value of the 
parameter at any given time. As long as the pointer lies within this 
range, the value of the parameter is considered essentially normal for 
the current conditions. If the pointer lies below the arc, then the 
value is significantly lower than expected, and if it is above the arc, 
then it is significantly higher than expected. Either case will 
generate a caution or warning condition, depending on the magnitude of 
the difference. Cautions or warnings will also be generated if the 
parameter enters a fixed caution or warning range for that parameter, 
even if the expected value also falls within those ranges for the 
current conditions. 

The column deviation format shown in Figure 5 [fig. B5] will require 
some additional explanation. The height of any bar in this display 
format does not depict the current value of the parameter it represents; 
rather, it depicts the difference between the current value and the 
expected value. A bar that extends below the centerline shows that the 
current value is lower than expected, while one that extends above the 
line shows that the current value is higher than expected. The 
additional two lines both above and below the centerline mark the 
regions of caution and warning alerts. A caution alert is generated 
whenever a difference of 10% or more exists between the current and 
expected values for a parameter, and a warning alert is generated 
whenever a difference of 15% or more exists. In addition, if a 
parameter value exceeds either the caution or warning limits for that 
parameter (whether it matches the expected value or not), the bar height 
reflects the proximity to the appropriate limit. 

The expected values for all the formats are generated by a 
numerical model of the engine, and since this model cannot perfectly 
predict the engine output, it is normal for slight differences to exist. 
On the circular gauges and bargraphs, this means that the pointer will 
frequently be a little off-center with respect to the expected value 
range, while on the column deviation display it means that short bars 
either above or below the centerline may be present. 

Also, on the circular gauge and bargraph display formats both with 
and without the expected value range, labelling will normally appear for 
all gauges that are displayed. For the column deviation display format, 
however, no gauge labelling will appear except for those individual 
parameters that are either significantly different from their nominal 
value or in a fixed caution or warning region. However, each display 
area for column deviations is labelled with the name of the appropriate 
aircraft subsystem. All gauge labels were included in Figure 5 [fig. B5] 
to let you see where the various parameters are displayed. 

To summarize, there are three different display element types: 
circular gauges, bargraphs, and column deviations. Two additional 
display formats were created by adding an expected value range to the 


52 


circular gauges and bargraphs, which makes a total of five display 
formats: (1) circular gauges; (2) bargraphs; (3) circular gauges with 

expected value range; (4) bargraphs with expected value range; and (5) 
column deviations. The test will be conducted in five sections, one for 
each of the above formats. 


Displ av-by-Exception Styles 


The display formats as shown on the previous pages all use the same 
style: (1) all parameters for all systems are always displayed. The 

remaining styles are as follows: (2) primary engine parameters are 

always displayed, but parameters for other systems are only displayed 
when those systems are affected by a fault; (3) all parameters of the 
same type are displayed for affected systems when one of those 
parameters is out-of-tolerance (e.g. all hydraulic pressure gauges are 
displayed even if only one of them is low) ; and (4) only out-of- 
tolerance parameters are displayed for affected systems. An example of 
each of these remaining display-by-exception styles, using the circular 
gauge format with expected value range, is shown in Figures 6 through 8 
[figs. B6 through B8] . 


The values used for all parameters are representative of a Boeing 
737-200 while in cruise flight at about FL330, at about Mach 0.7, on a 
near standard day. To aid your memory, the following list gives the 
normal range of values for each parameter under these conditions: 


Engines: EPR: 1.65 to 1.80 

Ni : 79.0 to 83.0 %RPM 

EGT: 320 to 340 °C 

N 2 : 81.0 to 84.0 %RPM 

FF : 1.89 to 2.12 x 1000 pph 

Oil: Pres: 42 to 52 psi 

Temp: 70 to 90 °C 

Quan: 3.0 to 4.0 gal 

Hydraulic: Pres: 2950 to 3050 psi 

Temp: 52 to 62 °C 

Quan: 3.1 to 3.3 qts 


Gen Oil Temp: 

70 

to 

90 

°C 

Volts : 

113 

to 

117 

V 

Frequency : 

398 

to 

402 

Hz 


Fuel: Quan: 5.50 to 6.50 x 1000 lbs 

Temp: —20 to +20 °C 


Whenever abnormal parameter values are displayed, they will differ 
significantly from the expected normal value and will fall outside of 
the ranges given above. 
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Test Sequence 

To display the first aircraft system state on the computer screen, 
press the space bar on the keyboard. As soon as your have determined 
which systems are affected and the status of the parameters for those 
systems, press the space bar again. This erases the display and records 
how long you were looking at it. You will then state, verbally: (1) 

which aircraft systems were affected and (2) which parameters in those 
systems were abnormal and what the status of each abnormal parameter was 
{low warning, low caution, normal, high caution, or high warning) . 

After you are done, press the space bar one more time. This records how 
long it took for you to provide the necessary information, (Some cases 
will take longer than others because of the number of affected systems 
and abnormal parameters.) After your answers are recorded by the 
researcher, press the space bar again to display the next aircraft 
system state. You will be notified when new formats and exception 
styles are about to be used. The entire study consists of 140 displays, 
which should take about two hours to complete. 

While participating in this study, please place your emphasis on 
accuracy rather than speed. It's better to take a little longer and get 
everything right than to rush and make mistakes. And remember, this 
study is not testing you , it's testing the displays . 

Trainin g Se quence 

Before beginning the actual test runs, you will participate in a 
brief training exercise to familiarize yourself with the different 
display formats and presentation styles. Don't worry at all about 
trying to go fast - just get comfortable with what you're going to see 
and do during the test. And if you have questions at all - PLEASE 
ASK! 
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TRIAL RESPONSE FORM 


Subject Number : Trial : 


Affected Systems: Enqines 

Oil 

Hydraulics 

Electrical 

Fuel 

Affected Parameters: 






Enginel : Epr 
Engine2 : Epr 
Oil 1 : Pres 

Ni 


Egt 

N? 

FF Vib 


Ni 


Egt 

N9 _ 

FF Vib 


Temp 


Quan 




Oil2 : Pres 

Temp 


Quan 




HydraulicA : 
HydraulicB : 
Hydraulics : 
Electricall : 
Elect rical2 : 
Fuel: QuanL 

Rpcnnn c 9 

Pres 

Temp 


Quan 



Pres 

Temp 


Ouan 



Pres 

Temp 


Quan 



GenOilT 

Volt 

Freq 



GenOi IT 


Volt 

Freq 



OuanC 

OuanR Temp 









r\vZr u u 1 1 o c • 



Trial : 




Affected Systems: Enqines 

Oil 

Hydraulics 

Electrical 

Fuel 

Affected Parameters: 







XT' i-v /"T "i n ps 1 

: Epr 

Ni 

Egt 

n 2 

FF 

Vib 

Hi I1U JL 1 1 C 1 
TTnrri 

: Epr 

Ni 

Egt 

n 2 

_ FF 

Vib 

Hj 1 1 y j- i 

Oill : 
Oi 12 : 

Pres 

Pres 

Temp 

Temp 

Quan 

_ Quan 

— 




HydraulicA: Pres Temp Quan 

HydraulicB : Pres Temp Quan 

Hydraulics: Pres Temp Quan 

Electricall: GenOilT Volt Freq 

Electrical2 : GenOilT Volt Freq 

Fuel: QuanL QuanC QuanR Temp 


Response : 


SUBJECTIVE EVALUATION 
PARAMETER PRESENTATION STUDY 


For each of the following twelve guestions (1-12), please make a mark 
inside the block that best describes your answer. Do not mark on the 
block dividers. 

Definitions: very slowly: extremely large amount of time required 

neither: neither particularly quickly nor slowly 
very quickly: little or no time required 

very difficult: intense mental effort required 
neither: neither particularly easy nor difficult 
very easy: little or no mental effort required 

The blocks in between the extremes and the middle of each scale indicate 
"somewhat . " For example, the block between "very slowly" and "neither" 
would correspond to "somewhat slowly," while the block between "neither" 
and "very easy" would correspond to "somewhat easy." 

NOTE: The following four questions (1-4) refer only to the three 

types of display elements (circular gauges, bargraphs, 
and column deviations) , regardless of whether or not expected 
value ranges were included or which display-by-exception 
presentation style was used. 

1. How quickly could you detect out-of-tolerance parameters? 


i) With circular gauges 

I I I I I J 


very 

slowly 


neither 


very 

quickly 


ii) With bargraphs 

llll 


very 

slowly 


neither 


very 

quickly 


iii) With column deviations 


very 

slowly 


neither 


very 

quickly 


2. How easy was it to remember which parameters were out-of-tolerance? 


i) With circular gauges 


ii) With bargraphs 


very 

difficult 


neither 


very 

easy 


very 

difficult 


neither 


very 

easy 


iii) With column deviations 


very 

difficult 


neither 


very 

easy 
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3 . 


How quickly could you determine whether the out-of-tolerance 
parameters were too high or too low? 


i) 

| 

With circular 

l 1 1 

gauges 

ii) 

1 

With bargraphs 

J J J 1 

1 1 

very 

slowly 

neither 

very 

quickly 

very 

slowly 

neither 

very 

quickly 

iii) 

With column 

deviations 




very 

slowly 

neither 

very 

quickly 





4 . How easy was it to remember whether the out-of-tolerance parameters 
were too high or too low? 


i) With circular gauges 


very neither very 

difficult easy 


ii) With bargraphs 


very neither very 

difficult easy 


iii) With column deviations 


very neither very 

difficult easy 


NOTE: The following four questions (5-8) refer only to the addition of 
expected value ranges on the displays, regardless of the display 
element type or which display-by-exception presentation style was 
used . 


5. How much more quickly or slowly could you detect out-of-tolerance 
parameters when expected value ranges were added? 


much more neither much more 
slowly quickly 


6. How much easier or more difficult was it to remember which 

parameters were out-of-tolerance when expected value ranges were 
added? 


much more 
difficult 


neither 


much 

easier 


7. How much more quickly or slowly could you determine whether the out- 
of-tolerance parameters were too high or too low when expected value 
ranges were added? 


much more neither much more 
slowly quickly 


8. How much easier or more difficult was it to remember whether the 

out-of-tolerance parameters were too high or too low when expected 
value ranges were added? 


much more neither much 

difficult easier 


NOTE: The following four questions (9-12) refer only to the four 

different display-by-exception presentation styles, regardless of 
the type of display element or whether or not expected value 
ranges were included. To refresh your memory, a brief description 
of each of the presentation styles is printed below. 


STYLES : 


1} all parameters for all systems are always displayed; 

2) primary engine parameters are always displayed, but 
parameters for other systems are only displayed when those 
systems are affected by a fault; 

3) all parameters of the same type are displayed for affected 
systems when one of those parameters is out-of-tolerance 
(e.g. all hydraulic pressure gauges are displayed even if 
only one of them is low) ; 

4) only out-of-tolerance parameters are displayed for affected 
systems . 


9. How quickly could you detect out-of-tolerance parameters? 


i) 

1 

With style 1 

1 1 

ii) 

1 

With style 2 
1 1 1 

1 1 

very 

slowly 

neither 

very 

quickly 

very 

slowly 

neither 

very 

quickly 

iii) 

With style 3 

1 1 

iv) 

With style 4 

1 1 

very 

slowly 

neither 

very 

quickly 

very 

slowly 

neither 

very 

quickly 
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10 . 


How easy was it to remember which parameters were out-of-tolerance? 


i) With style 1 

1 1 __l 1 1 J 

very neither 

very 

di f f icult 

easy 

iii) With style 3 

i i i i i 

! 

very neither 

very 

difficult 

easy 


11. How quickly could you determine 
parameters were too high or too 

i) With style 1 


very 

neither 

very 

slowly 


quickly 

iii) 

| 

With style 3 

1 1 l_ 

J J 

very 

slowly 

neither 

very 

quickly 


12. How easy was it to remember whether 
were too high or too low? 


i) With style 1 

very 

neither 

very 

di f f icult 

easy 

iii) 

1 

With style 3 

1 1 1 L 


very 

neither 

very 

difficult 

easy 


ii) With 

1 1 

style 2 
1 1 1 

1 

very 

neither 

very 

difficult 


easy 

iv) With 
1 1 

style 4 

J 1 L 

i 



very neither 

very 


di f f icult 

easy 

whether 

the out -of -tolerance 


low? 

ii) With style 2 

1 1 1 1 L 



very 

slowly 

neither 

very 

quickly 

iv) 

1 

With style 4 
1 1 1 

J 1 

very 

slowly 

neither 

very 

quickly 

out-of ■ 

-tolerance parameters 


ii) With style 2 


1 1 1 J ^ 1 

very neither 

very 

difficult 

easy 

iv) With style 4 

1 1 1 1 __L 

1 

very neither 

very 

difficult 

easy 
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For the following ranking questions (13-14), please write the 
appropriate number in the blank beside each selection you make. 

STYLES: 1) all parameters for all systems are always displayed; 

2) primary engine parameters are always displayed, but 
parameters for other systems are only displayed when those 
systems are affected by a fault; 

3) all parameters of the same type are displayed for affected 
systems when one of those parameters is out-of-tolerance 
(e.g. all hydraulic pressure gauges are displayed even if 
only one of them is low) ; 

4) only out-of-tolerance parameters are displayed for affected 
systems . 


13. 


P lease select the three combinations of display format, expected 
value range, and presentation style that you like the best (l=best, 
2=next best, 3=third best). Pick only three. 


circular gauges 
circular gauges 
circular gauges 
circular gauges 
bargraphs 
bargraphs 
bargraphs 
bargraphs 
circular gauges 
circular gauges 
circular gauges 
circular gauges 
bargraphs 
bargraphs 
bargraphs 
bargraphs 
column deviations 
column deviations 
column deviations 
column deviations 


no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 


style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
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STYLES : 


14 . 


1) all parameters for all systems are always displayed; 

2) primary engine parameters are always displayed, but 
parameters for other systems are only displayed when those 
systems are affected by a fault; 

3) all parameters of the same type are displayed for affected 
systems when one of those parameters is out-of -tolerance 
(e.g. all hydraulic pressure gauges are displayed even if 
only one of them is low) ; 

4) only out-of-tolerance parameters are displayed for affected 
systems . 


Please select the three combinations of display format, expected 
value range, and presentation style that you liked the least 
( l=worst , 2=next worst, 3=third worst) . Pick only three. 


circular gauges 
circular gauges 
circular gauges 
circular gauges 
bargraphs 
bargraphs 
bargraphs 
bargraphs 
circular gauges 
circular gauges 
circular gauges 
circular gauges 
bargraphs 
bargraphs 
bargraphs 
bargraphs 
column deviations 
column deviations 
column deviations 
column deviations 


no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
no expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 
expected value range 


style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 
style 1 
style 2 
style 3 
style 4 



For each of the following four questions (15-18) , please include at 
least one sentence for each requested response. 


15. Please give the reasons for choosing your three most preferred 
combinations from question 13. 


16. Please give the reasons for choosing your three least preferred 
combinations from question 14. 


17 . How could the format /presentation style combination that you most 
preferred be improved further? 


18. What situations might exist in which a format /presentation style 
combination other than your most preferred one may be useful or 
necessary? Which one(s) ? 
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19. Please record any other comments, suggestions, or criticisms you may 
have about any of the display element types, the inclusion of 
expected value ranges, the display-by-exception presentation styles, 
or the way the experiment was conducted. 















Figure B2. [Bow tie] bargraphs 
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Figure B5. Column deviations. 
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